LLM Ops
Large Language Model Ops (LLMOps) encompasses the practices, techniques and tools used for the operational management of large language models in production environments.
See:
Resources
- Data-centric MLOps and LLMOps | Databricks
- LLMOps: Operationalizing Large Language Models | Databricks
- LLMops 101: A Detailed Insight into Large Language Model Operations | Medium
- LLMs-based systems evaluation: A tremendous pillar of LLMOps | by Wael SAIDENI | Medium
Benchmarks and evaluation
See LLMs evaluation
LLMs in production
- Creación de aplicaciones de IA generativa con modelos de base – Amazon Bedrock – AWS
- https://docs.bentoml.org/en/v1.1.11/quickstarts/deploy-a-transformer-model-with-bentoml.html
- How to deploy Meta Llama models with Azure Machine Learning studio - Azure Machine Learning | Microsoft Learn
Courses
Code
- #CODE GitHub - AgentOps-AI/agentops
- Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks like CrewAI, Langchain, and Autogen
- AgentOps
- Agent Tracking with AgentOps | AutoGen
- AgentOps, the Best Tool for AutoGen Agent Observability | AutoGen
- #CODE Langsmith langchain-ai/langsmith-cookbook (github.com)
- #CODE Opik - Open-source end-to-end LLM Development Platform
- Confidently evaluate, test and monitor LLM applications.
- Opik by Comet | Opik Documentation
Serving LLMs, VLMs
- #CODE vllm-project/vllm - high-throughput and memory-efficient inference and serving engine for LLMs